Skip to content

refactor: replace 2 DB-busy retry loops with tenacity (#1132) - #1282

Merged
axisrow merged 2 commits into
mainfrom
ao/tg_content_factory_5863f66be3-64/tenacity-db-busy-retry
Jul 27, 2026
Merged

refactor: replace 2 DB-busy retry loops with tenacity (#1132)#1282
axisrow merged 2 commits into
mainfrom
ao/tg_content_factory_5863f66be3-64/tenacity-db-busy-retry

Conversation

@axisrow

@axisrow axisrow commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Part of #1132 (axis 2/7 of #1130). Registry: #782 — retry batch, DB-busy group. Follows up #1198 (Counter/base64 safe subset), which explicitly deferred all retry findings for per-case owner review.

What

Replaces the two genuine hand-rolled retry loops among the 14 detector findings with tenacity.AsyncRetrying (already a direct dependency; same style as error_recovery_service.py / production_limits_service.py). The other 12 findings are poll/pacing/work-distribution loops — adjudication table below, no code changes.

1. Database._with_busy_retry (src/database/facade.py)

SQLite-busy retry on the shared write path. Behaviour preserved exactly:

  • fixed delay ladder from self._busy_retry_delays_sec via wait_chain(*(wait_fixed(d) ...)) — no jitter, no exponent;
  • attempts = len(delays) + 1 via stop_after_attempt;
  • non-busy OperationalError re-raised immediately (retry_if_exception(_is_retryable_busy_error));
  • same per-retry and final warning logs; final failure raises DatabaseBusyError chained (from) to the last busy error.

2. TelegramCommandDispatcher._update_command_safely (src/services/telegram_command_dispatcher.py)

Unbounded exponential backoff on DatabaseBusyError. Behaviour preserved exactly:

  • ladder INITIAL, 2×, 4×, … capped at MAX via wait_exponential(multiplier=INITIAL, max=MAX) — bit-identical float sequence (doubling is exact in binary);
  • retry_busy=False → single attempt, busy swallowed with the same log line;
  • non-busy exceptions swallowed with the same log line (status updates must never kill the run loop);
  • sleep=asyncio.sleep keeps the existing test seam — suite patches of mod.asyncio.sleep still observe the retry sleeps;
  • CancelledError still propagates (tenacity re-raises BaseExceptions, except Exception doesn't catch them) — test_run_loop_cancelled_reraises_when_update_busy stays green.

tenacity trap discovered (for the #782 registry)

AsyncRetrying awaits the result of fn(), so fn must be an async callable. _with_busy_retry receives a sync Callable returning a coroutine (lambda: begin_immediate(...)). Without the _wrap_async adapter the coroutine was never awaited → BEGIN IMMEDIATE never ran → transaction() hit cannot COMMIT - no transaction is active, leaking out as PytestUnraisableExceptionWarning (which is an error under the project's filterwarnings = ["error"], so it silently broke every DB write and red the whole suite). The adapter is one line (async def _runner(): return await action()); the dispatcher path is unaffected because its _update is already async def. Worth a line in #782 so future retry→tenacity conversions don't repeat it.

Retry/lifecycle check (#958 trap)

Both loops are local SQLite retries — no network, no billing. SQLITE_BUSY means the statement did not execute, so a retry cannot double-apply a write. tenacity adds no hidden extra retry layer here (no SDK involved); attempt counts and delay sequences are pinned by tests.

TDD / behavioural equivalence

7 characterisation tests were added first and verified green on the old implementation, then the swap landed with the tests untouched:

  • test_database.py: exact sleep ladder on exhaustion; result returned after transient busy (slept exactly delays[0]); non-busy OperationalError raise-through with zero sleeps; DatabaseBusyError.__cause__ is the last busy error.
  • test_telegram_command_dispatcher.py: exponential ladder doubles and caps (computed from module constants); retry_busy=False single-shot without sleep; generic error swallowed without retry.

Detector / baseline

  • Findings for both converted functions are gone; --fail-on-new green.
  • Baseline refreshed (13 → 12 entries): −2 fixed; pure path renames folded in (agent/manager.pyagent/backends/_stream.py, telegram/collector.pytelegram/collector_mixins/* — same findings, files moved by earlier refactors); +1 genuinely new finding recorded (web/scheduler/context.py:236, see table).

Adjudication of the remaining 12 findings (proposal — owner decides, #782)

Finding Class Proposal
agent/backends/_stream.py:463 _await_with_countdown deadline-extension ticker for streaming UX — no operation is ever re-attempted keep (false positive)
services/task_handlers/stats.py handle_stats_all worker-pool batch loop with flood-defer; retries delegated keep (FP, same class as pool_dialogs #1114)
telegram_command_dispatcher.py _run_loop infinite service poll loop (claim → dispatch); sleeps are idle pacing keep
unified_dispatcher.py _run_loop same keep
collector_mixins/collection.py collect_all_channels iterates different channels with inter-channel stagger; errors break/continue, never re-attempt keep (FP)
collector_mixins/stats.py collect_all_stats worker-pool with defer keep (FP)
pool_dialogs.py warm_all_dialogs, leave_channels already adjudicated FP — #1114, AST guards in #1149/#1176 keep (guarded)
pool_dialogs.py delete_dialogs same pattern as leave_channels: retry delegated to run_with_flood_wait_retry, loop iterates different dialogs keep (FP)
web/bootstrap.py _retry_telegram_pool_until_connected condition-driven reconnect loop with cooperative-shutdown checks between waits; tenacity is exception-driven — conversion would bury the shutdown logic keep
web/search/handlers.py _telegram_search_via_worker poll-until-deadline for a DB command status, not a failure retry keep (FP)
web/scheduler/context.py:236 _dedupe_recent_unavailability_events (NEW, counter) group-by with two aggregates (count + max timestamp); collections.Counter covers only the count and would split a single-pass aggregation into two structures propose keep

Registry row (#782 journal):

Дата Находка (файл) Стандартная замена Решение PR/issue
2026-07-10 database/facade.py _with_busy_retry; telegram_command_dispatcher.py _update_command_safely (DB-busy retry) tenacity.AsyncRetrying заменено (этот PR) this PR
2026-07-10 остальные 12 retry/counter-находок предложение «оставить» (таблица выше), финальное слово за владельцем this PR

Not merging — owner reviews (retry→library is in the dual-review category per the task brief).

🤖 Generated with Claude Code

Part of #1132 (axis 2/7 of #1130). Registry: #782 — retry batch, DB-busy
group. Follows up #1198 (Counter/base64 safe subset), which deferred all
retry findings for per-case owner review.

Replaces the two genuine hand-rolled retry loops among the 14 detector
findings with tenacity.AsyncRetrying (already a direct dependency; same
style as error_recovery_service.py / production_limits_service.py). The
other 12 findings are poll/pacing/work-distribution loops — adjudication
table in the PR, no code changes.

1. Database._with_busy_retry (src/database/facade.py) — SQLite-busy retry
   on the shared write path. Behaviour preserved exactly:
   - fixed delay ladder from self._busy_retry_delays_sec via wait_chain
     (no jitter, no exponent);
   - attempts = len(delays)+1 via stop_after_attempt;
   - non-busy OperationalError re-raised immediately;
   - same per-retry/final warning logs; final failure raises
     DatabaseBusyError chained to the last busy error.

2. TelegramCommandDispatcher._update_command_safely — unbounded
   exponential backoff on DatabaseBusyError. Behaviour preserved exactly:
   - ladder INITIAL, 2x, 4x... capped at MAX via wait_exponential;
   - retry_busy=False -> single attempt, busy swallowed;
   - non-busy exceptions swallowed (status updates never kill the loop);
   - sleep=asyncio.sleep keeps the existing test seam; CancelledError
     still propagates (tenacity re-raises BaseException).

tenacity trap discovered (#1132): AsyncRetrying awaits the result of
fn(), so fn must be an async callable. _with_busy_retry receives a
SYNC Callable returning a coroutine (lambda: begin_immediate(...));
without the _wrap_async adapter the coroutine was never awaited, the
BEGIN never ran, and "cannot COMMIT - no transaction is active" leaked
out via PytestUnraisableExceptionWarning (errors under filterwarnings=
["error"]). Recorded for the #782 registry.

Retry/lifecycle check (#958 trap): both loops are local SQLite retries
(SQLITE_BUSY = statement did NOT execute), so a retry cannot double-apply
a write. No network, no billing, no SDK retry layer added.

TDD: 7 characterisation tests added first (green on old impl), then the
swap landed with the tests untouched:
- test_database.py: exact sleep ladder on exhaustion; result after
  transient busy; non-busy OperationalError raise-through (zero sleeps);
  DatabaseBusyError.__cause__ is the last busy error.
- test_telegram_command_dispatcher.py: exponential ladder doubles+caps;
  retry_busy=False single-shot; generic error swallowed.

Detector: findings for both functions gone; --fail-on-new green.
Baseline refreshed (13 -> 12): -2 fixed; pure path renames folded in
(agent/manager.py -> agent/backends/_stream.py, telegram/collector.py
-> collector_mixins/*); +1 new finding recorded
(web/scheduler/context.py:236, counter with two aggregates -> propose
keep).

Co-Authored-By: Claude <noreply@anthropic.com>
@axisrow
axisrow force-pushed the ao/tg_content_factory_5863f66be3-64/tenacity-db-busy-retry branch from 2a8a69a to 7a67287 Compare July 27, 2026 14:37
@axisrow

axisrow commented Jul 27, 2026

Copy link
Copy Markdown
Owner Author

🔍 Local review (cycle 1)

Reviewed locally (/review + Codex companion sol/xhigh), no bots pinged. Round bound to head 7a67287e.

Codex verdict: approve (0 findings). Runtime-проверено: _wrap_async корректно await'ит action → реальные writes и BEGIN → DML → COMMIT выполняются; retry counts, delay ladders, exception chaining, single-shot mode, cancellation propagation, non-busy handling — эквивалентны старому поведению. (pytest не запущен в sandbox Codex — read-only tmpdir; локально 743 теста зелёные.)

/review verdict: no blocking issues (2 minor):

Verdict Reviewer Finding Location
SKIP claude _log_retry читает tenacity-internal retry_state.next_action.sleep — недокументированное поле; при апгрейте tenacity деградирует в 0.0 (есть is not None-гард → лог-only, не краш, не поведение retry) src/database/facade.py:267
SKIP claude нет явного теста «_wrap_async выполняет action» — но test_busy_retry_returns_action_result_after_transient_busy уже неявно защищает удаление адаптера (возвращает sentinel, не coroutine); стоит отметить связь в комментарии tests/test_database.py:241

No FIX, no UNVERIFIED — cycle 1 clean. Не мержу (local mode — merge за владельцем).

#1282)

Cycle-review cleanup (local, cycle 1 — 2 SKIP findings, no FIX):

1. src/database/facade.py _log_retry: compute the retry delay from our own
   delay ladder (delays[attempt_number-1]) instead of reading the
   undocumented tenacity-internal retry_state.next_action.sleep. Same logged
   value (verified: attempt_number is the just-failed 1-based attempt, after
   which we sleep delays[attempt_number-1]); no longer fragile to a tenacity
   upgrade removing/renaming next_action. Guarded idx bounds → 0.0 fallback.

2. tests/test_database.py: docstring on test_busy_retry_returns_action_result_
   after_transient_busy now states it also guards _wrap_async (returns sentinel
   only if the coroutine was actually awaited — without the adapter tenacity
   returns an un-awaited coroutine and `result is sentinel` fails).

No behaviour change; ruff clean; 641 affected tests green.

Co-Authored-By: Claude <noreply@anthropic.com>
@axisrow

axisrow commented Jul 27, 2026

Copy link
Copy Markdown
Owner Author

📋 Review summary — all cycles

Cycle Reviewer Finding Verdict Resolution
1 codex (runtime probe: _wrap_async awaits, BEGIN→DML→commit, ladder/counts/cause/single-shot/cancel/non-busy equivalent) approve (0 findings)
1 claude _log_retry reads tenacity-internal next_action.sleep (undocumented, fragile to upgrade) SKIP → applied c2 Fixed in fdf3a1b (compute delays[attempt_number-1] instead)
1 claude no explicit test that _wrap_async executes action (implicit guard exists) SKIP → applied c2 Clarified in fdf3a1b (test docstring names the invariant)
2 codex (boundary probe: delays[attempt_number-1] exactly matches former next_action.sleep; terminal attempt raises RetryError without before_sleep; identical sequences, no behavior change) approve (0 findings)
2 claude cleanup commit correct & safe clean (0 findings)

Totals: 0 FIX, 2 SKIP (both applied as cleanup in fdf3a1be), 0 UNVERIFIED.

Review binding: head fdf3a1be, base main @ 53001148. Local mode — merge is the owner's call.

@axisrow
axisrow merged commit e8f5f4c into main Jul 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant